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The U.S. Census Bureau thanks the Census Scientific Advisory Committee for its 
recommendations. We are responding to the committee recommendations submitted during 
its March 28-29, 2019, meeting. 

Your expertise is important to ensuring that the Census Bureau continues to provide relevant 
and timely statistics used by federal, state, and local governments, as well as business and 
industry, in an increasingly technologically-oriented society. 
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Recommendations and Comments to the Census Bureau 
from the Census Scientific Advisory Committee 
Spring 2019 Meeting 


To: Steven D. Dillingham 

Director 

U.S. Census Bureau 

From: Allison Plyer 
CSAC Chair 

March 28, 2019 


I. Update on the 2020 Census 

CSAC was pleased to hear that the Census Bureau has already done a great deal of the risk assessment 
and risk mitigation analysis in response to the concerns that were raised by CSAC. For example, CSAC is 
reassured that Census is already cooperating with other federal agencies and with social media 
companies to thwart potential cyber-attacks on Census cyber-infrastructure and to cooperate in the 
monitoring of social media for potential manipulation by foreign actors on response levels. 


CENSUS RESPONSE: 

The Census Bureau agrees with the statement and continues to prioritize risk assessment and mitigation 
analysis in response to the concerns raised by CSAC. 


II. Possible Addition of the Citizenship Question 

The Supreme Court is expected to decide the issue of whether or not the Census may add the citizenship 
question by June of this year. In the event that the Supreme Court rules in favor of adding the 
citizenship question, CSAC requests that Census provide additional information at the fall 2019 CSAC 
meeting as described below: 


1. A March 2019 Harvard study entitled "Estimating the Effect of Asking about Citizenship on the 
U.S. Census: Results from a Randomized Controlled Trial" (available at: 
https://shorensteincenter.org/estimatine-effect-asking-citizenship-u-s-census/ ) provides 
credible quantitative evidence that the addition of a citizenship question would "significantly 
increase the percent of questions skipped, with particularly strong effects among Hispanics, and 
makes respondents less likely to report having members of their household who are of Hispanic 
ethnicity." Further, this study concluded that the addition of the citizenship question would 
have reduced "the number of Hispanics reported in the 2010 Census by approximately 4.2 
million." However, the researchers did not know where the Citizenship Question would be 
asked on the 2020 Census survey, and therefore, randomly rotated where it appeared on their 
survey instrument. It is known that there are many subtleties associated with adding a question 
to a survey. For example, where the new question is placed on the census form can affect 
response rates on other questions, and this new question is likely to affect self-identification by 
racial and ethnic group. CSAC would like to understand, in more detail, where the question will 
be placed on the 2020 Census questionnaire and how Census expects to assess the impact of the 
addition of the citizenship question on the response rate to the other questions that will be 




asked on the 2020 Census form. CSAC would like a report-out on the results of the summer 2019 
survey testing the impact of the citizenship question at our fall 2019 meeting. 


CENSUS RESPONSE: 

In Department of Commerce v. New York, No. 18-966 (June 27,2019), the Supreme Court 
determined the Department's explanation to include a citizenship question on the 2020 Census 
insufficient to support the Department's decision. On July 2, the Census Bureau initiated printing 
of the 2020 questionnaires without a citizenship question and continues to do so in accordance 
with the "Executive Order on Collecting Information about Citizenship Status in Connection with 
the Decennial Census" issued July 11, 2019. Given the recent turn of events, CSAC's request is 
now moot. 


2. The Bureau expects to gather data during the 2020 Census that would be useful in determining 
how to adapt its editing, imputation and other protocols used post-processing to reflect the 
differential impact of census non-response across the different states to inform 
reapportionment. CSAC requests more detail on how Census intends to deal with this issue. 


CENSUS RESPONSE: 

The Census Bureau accepts the request to provide more detail about how the imputation and 
other protocols in the post-processing reflect the census non-response. 

Following data collection in past censuses, the Census Bureau has not processed the data 
differently from state to state according to the different levels of self-response. This will remain 
the same for the 2020 Census. 

After data collection is completed, the Census Bureau will use count imputation for unresolved 
addresses. The procedure is executed independently for each state. The Census Bureau would 
be happy to provide more information about the count imputation procedure at a future 
meeting. 

We want to be clear that the Census Bureau does not make any adjustments to the final census 
counts and reapportionment results based on the results of the Post-Enumeration Survey, 
Demographic Analysis, or any other data collection. 


3. The addition of this question could significantly increase the volatility of the overall non¬ 
response rate to the 2020 Census. It is possible that the full scale of likely non-responsiveness 
may only reveal itself early in 2020, depending on the degree to which a large-scale non- 
compliance movement were to get underway. So, if the question is added. Census needs to fully 
prepare to pivot quickly in early 2020 to whatever measures would need to be taken if it 
appears that non-response will be (much) higher than planned for. 
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CENSUS RESPONSE: 


In Department of Commerce v. New York, No. 18-966 (June 27,2019), the Supreme Court 
determined the Department's explanation to include a citizenship question on the 2020 Census 
insufficient to support the Department's decision. On July 2, the Census Bureau initiated printing 
of the 2020 questionnaires without a citizenship question and continues to do so in accordance 
with the "Executive Order on Collecting Information about Citizenship Status in Connection with 
the Decennial Census" issued July 11,2019. 

We will continue to strive for the most accurate and complete count possible. To this end, we 
will persist in leveraging partners across the nation to spread awareness of the importance of a 
complete and accurate count during the multiple phases of our communications campaign. The 
campaign begins with an awareness phrase, during which time messaging is focused on 
educating people about the importance of participating in the 2020 Census. This is followed by a 
motivation phase, when messaging is focused on encouraging people to "self-respond to the 
2020 Census". Next comes a reminder phrase, in which messaging is focused on "it is not too 
late to self-respond," and "cooperate with census takers who come to your door." 

The process of engaging partners is well underway. Forty-eight states (plus the District of 
Columbia and Puerto Rico) have either formed complete count commissions or are in the process 
of doing so, supplemented by thousands of complete count committees now operational at the 
local level. Hundreds of organizations have already agreed to be official 2020 Census partners as 
well. 


4. Administrative data holds the potential to provide more reliable citizenship data than adding the 
question to the 2020 Census survey. CSAC would like an update of the Census' August 2018 
study of alternative citizenship data sources now that the Census has acquired new 
administrative data (such as from US Citizenship and Immigration Services). 

CENSUS RESPONSE: 

The Census Bureau presumes the Committee is referring to the research conducted by the 
Research and Methodology Directorate to inform the Secretary's decision on citizenship. There 
was not a planned 2018 research project relating to alternative citizenship data, and the Census 
Bureau has no plans to conduct a formal study in the next year. 

The Census Bureau is negotiating with the State Department, the Department of Homeland 
Security, and other departments and agencies for relevant administrative records and working to 
receive test data to ensure our systems are able to read-in the files. We are happy to brief you 
on the status of the work at the fall meeting. 


5. With the addition of the citizenship question, the Census would need to decide on a Non- 
Response Follow-Up protocol that responds to the higher non-response rates that would be a 
likely result. In the event of limited NRFU resources, CSAC recommends that such follow-up be 
limited to priority data elements, excluding the Citizenship Question. CSAC would like a briefing 
on the protocol to be used. 

CENSUS RESPONSE: 

In Department of Commerce v. New York, No. 18-966 (June 27,2019), the Supreme Court 
determined the Department's explanation to include a citizenship question on the 2020 Census 
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insufficient to support the Department's decision. On July 2, the Census Bureau initiated printing 
of the 2020 questionnaires without a citizenship question and continues to do so in accordance 
with the "Executive Order on Collecting Information about Citizenship Status in Connection with 
the Decennial Census" issued July 11, 2019. 

We have no plans to change the design or contact strategies for the Nonresponse Followup 
operation. Given the recent turn of events, CSAC's request is now moot. 


III. End-to-End Tests 

It appears that the 2018 end-to-end tests have gone well. However, given how critical these tests are for 
ensuring the smooth functioning of the 2020 Census, CSAC would like to learn more in a one-way 
briefing regarding what weaknesses and vulnerabilities have been identified, and how Census is 
responding to these challenges. CSAC would be particularly interested to know about: 


1. Census' experience in the end-to-end test with using administrative data for NRFU or for post¬ 
processing imputation, 

CENSUS RESPONSE: 

The Census Bureau would be happy to provide a briefing on the findings from the 2018 End-to- 
End Census Test related to the use of administrative records for the NRFU operation. 

2. Lessons learned on training the enumerators on the new technology, and 

CENSUS RESPONSE: 

When training enumerators for NRFU using devices and the Enumeration instrument Field Data 
Capture (FDC), we learned that enumerators needed additional training, as well as hands-on 
experience in navigating through the instrument screens. In 2018, our training emphasized 
interviewing skills, not the technical skills needed to use the device. For 2020, we are adding an 
FDC skills module to our online training. Under this module, each trainee will walk through the 
entire interview path on their device as part of the classroom training role play exercises. For 
listing operations training, we primarily learned that we needed to simplify device login and 
provide a single sign-on approach with systems. We have implemented both of these updates. 

3. Lessons learned from the online responses, including paths followed by users, drop-off points, 
and share using mobile device by demographic group. 

CENSUS RESPONSE: 

The Census Bureau will be evaluating information gathered during the 2018 End-to-End Census 
Test about how users interacted with the Internet Self-Response instrument. When that 
evaluation is complete and available for release, we would be happy to provide a briefing on the 
data. We anticipate that we will complete the evaluation in the fall of 2019. 

4. An analysis of how self-response and self-response by internet correlate with tract-level 
indicators of computer and internet access newly available from the ACS. 
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CENSUS RESPONSE: 


The Census Bureau accepts this recommendation. We are planning to conduct this analysis. We 
would be happy to share the results with the CSAC. They are expected in fall of 2019. 


In addition to the recommendations above, we also believe that the "type of enumeration area" map is 
very helpful to local outreach planning. CSAC recommends that the Bureau publish the tract-level data 
on which tracts will receive the "internet-first" (i.e. invitation letter only) vs. "internet-choice" (i.e. a 
form with the first mailing) to help local groups tailor their messaging and outreach to different 
neighborhoods. 

CENSUS RESPONSE: 

This is a helpful recommendation, and we will release this information in late fall. 


CSAC requests that the Census Bureau consider the benefits of publishing video of the interface before 
the actual data collection to help community groups provide guidance on navigating the 2020 interfaces, 
particularly for hard-to-count groups. 

CENSUS RESPONSE: 

The Census Bureau will produce video language guides that walk respondents through the internet 
instrument. These videos will provide general information about the 2020 Census, a brief overview of the 
internet instrument, and a detailed view of the questions. These videos will be available in English and in 
59 non-English languages. 


Finally, CSAC requests that the Fall 2019 CSAC meeting agenda include a discussion of the evaluations 
and assessments that are planned for the 2020 Census coverage and quality, specifically the Post- 
Enumeration Survey, and the Demographic Analysis components of the Census Program for Evaluations 
and Experiments (CPEX) for 2020. We would appreciate a listing of all the planned CPEX components in 
the Bureau's written response. 

CENSUS RESPONSE: 

The Census Bureau would be happy to provide the Committee with a presentation covering the 
operational assessments and the 2020 CPEX. The final list of CPEX projects is still undergoing internal 
review and approval but can be provided to the Committee upon final approval. 


IV. Census Barriers, Attitudes and Motivators Survey and Focus Groups Final Report 

CSAC would like to commend the Census Bureau for their extensive and well-formulated research 
agenda dedicated to developing an effective communications plan for motivating self-response to the 
decennial census. The study design for CBAMS exemplifies this dedication, with both a quantitative 
survey component adhering to the highest standards of statistical practice, and qualitative focus groups 
providing context, support and illustrative information to support fuller understanding of the 
quantitative results. 

1. Census properly notes that CBAMS survey results (weighted estimates computed from those 
willing to complete a CBAMS survey) may be biased if CBAMS nonrespondents are in fact less 
willing to complete the decennial census than CBAMS respondents. Nonresponse bias is of 
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concern because CBAMS response behavior is plausibly correlated with decennial response 
propensity. While the nonresponse adjustment used for the CBAMS analysis reduces bias under 
a plausible assumed response model, CSAC recommends further empirical investigation to 
assess the quality of that assumed model. 

CENSUS RESPONSE: 

The Census Bureau accepts this recommendation. We plan to explore how CBAMS respondents' 
decennial census response behaviors differ from (1) their response behaviors stated in CBAMS 
and (2) the decennial census response behaviors of CBAMS non-respondents. (This also relates 
to Recommendation IV.2) 


2. CBAMS represents a unique opportunity to follow up on the decennial census response behavior 
for a randomized assignment of households into a CBAMS "treatment group" and a non-CBAMS 
"control group." All households in the treatment group received the CBAMS treatment, with 
some self-selecting into CBAMS respondents and some self-selecting into CBAMS non¬ 
respondents. All three groups can be followed to determine their actual decennial census 
response behavior, with the control group providing baseline response behavior in the decennial 
environment (including the communications campaign) and the two CBAMS groups providing 
response behavior before and after the decennial environment. CSAC recommends that Census 
undertake a research program to follow up with the CBAMS groups according to their decennial 
response behavior. 

CENSUS RESPONSE: 

The Census Bureau accepts these recommendations. Currently, we are exploring multiple 
options to utilize the 2020 CBAMS survey sample and findings to inform further research. 


3. CSAC recommends that if a second CBAMS is conducted after 2020, then at least some of the 
originally-selected CBAMS households should be selected for follow-up interviews. 

CENSUS RESPONSE: 

The Census Bureau accepts these recommendations. Currently, we are exploring multiple 
options to utilize the 2020 CBAMS survey sample and findings to inform further research. 


4. CBAMS found unfamiliarity, distrust, misconceptions, and concerns with the decennial census. 
These attitudes about Census may persist well past 2020. CSAC recommends that Census 
develop an ongoing monitoring effort of barriers, attitudes and motivators throughout the 
decade. 

CENSUS RESPONSE: 

The Census Bureau accepts this recommendation as understanding the public's pulse is critical to 
developing communications strategies for the intercensai years, including impact on American 
Community Survey (ACS) and other Census Bureau surveys. 
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V. Update on Integrated Partnership and Communications Program 

CSAC appreciates the preparations and research that have contributed to the development of the 
Integrated Partnership and Communications Program to date. The decennial census is an enormous 
undertaking requiring extensive field work. When respondents are motivated to participate, the field 
work is more cost effective, and the resulting data quality is better. 

CSAC has several recommendations for the Bureau to consider with the aim of increasing the 
effectiveness of the Integrated Partnership and Communications Program, while still carefully managing 
taxpayer dollars. 


1. CSAC recommends the Awareness phase begin in the fall of 2019 using social media targeting 
hard-to-count populations. This recommendation would not alter the traditional media plan, but 
it would leverage the availability of low-cost social media to partially implement one of the 
recommendations of the 2010 Census outreach program assessment. 

CENSUS RESPONSE: 

The Census Bureau agrees that certain, hard-to-count populations need earlier, more robust 
outreach. For that reason, we formally started our early education efforts in October 2018. At 
this moment, we have a very robust social media outreach with content updated on a monthly 
basis. However, your suggestion of paid social media advertising merits consideration, and 
Census Bureau leadership is currently evaluating its potential implementation. 

2. CSAC recommends evaluating advertising during the early half of the Motivation phase and 
adjusting ad selection so that the most effective ads are used more intensely during the latter 
part of the Motivation phase and during the Reminder phase. 

CENSUS RESPONSE: 

Thank you for your recommendation. The Census Bureau currently is planning its campaign 
optimization program, which is closely related with our Motivation and Reminder phases. We 
will be looking at several data sources, including response rates, to make adjustments in our paid 
advertising, but also in other efforts, such as partnerships, social media engagements, and media 
outreach. 


3. CSAC recommends that the Census Bureau use rapid response and nimble communications 
capabilities to respond to local communications crisis that can arise during the combined 
Census-Election Year. Specifically, Census should use and empower local partnership specialists 
to monitor local social media and to be able to respond to local communications crisis. 

CENSUS RESPONSE: 

Rapid response is a key part of our campaign through our campaign optimization efforts in 
which we will be using all campaign components, such as paid advertising, partnerships, and 
media outreach, to find quick solutions to specific response rate issues and local communications 
crises. Our local partnership specialists will be a vital element of these efforts, and social media 
monitoring and crisis management are included in their training program. However, as 
explained during the meeting, individual social media accounts are against Census Bureau social 
media policies and are not envisioned as part of the local partnership specialists tools. 
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4. CSAC recommends that the Bureau negotiate agreements with social media platforms to 
promote the Census. For example, there could be a Google Doodle or a Facebook status update 
or digital sticker that adds an "I filled in my Census" overlay over the person's profile pic. 
Another example could be partnering with Amazon so that Alexa reminds everyone to 
participate on Census Day. 

CENSUS RESPONSE: 

Team Y&R and the Census Bureau have discussed this possibility as part of both our paid 
advertising and national partnerships efforts, and we feel very positive of the possibility of 
implementing this idea. We will keep you posted. 

5. CSAC recommends that the Bureau explore ways to provide immediate feedback after self¬ 
response, such as viewing current response rates in their location or in competing locations. 

CENSUS RESPONSE: 

The Census Bureau will be providing daily updates of census self-responses during the 
enumeration period. The self-response rates will be provided at the tract level for all tracts with 
more than 20 households. The map and associated datafile will be updated once a day, 7 days a 
week from mid-March to mid-May and once a day, 5 days a week from mid-May to the end of 
July. 

6. CSAC recommends partnering with national privacy groups to aid in communicating that Census 
2020 participation is important for the functioning of our democracy and does not compromise 
the privacy of respondents nor is respondents' data shared with other agencies. Ensure that the 
messaging addresses the concerns raised in the CBAMS focus groups. 

CENSUS RESPONSE: 

We concur and efforts are underway. Our communications research determined that privacy 
and confidentiality are key themes that need to be addressed through our communications and 
partnership programs. Messaging has been developed and was tested through online panels, 
focus groups, and community representative reviews. The final messaging will be revised when 
the data has been analyzed. We can update the group on these findings at our next meeting. 

7. CSAC requests an update of the Integrated Partnership and Communications Program at the fall 

2019 CSAC meeting. 

CENSUS RESPONSE: 

The Integrated Partnership and Communications (IPC) program appreciates your interest and 
accepts this recommendation. 

8. CSAC recommends that the Bureau budget the necessary resources and secure the 
competencies necessary to evaluate the effectiveness of the communications plan after the 

2020 Census. 
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CENSUS RESPONSE: 


The Census Bureau accepts this recommendation and confirms that is currently evaluating how 
to assess the I PC program at the end of2020 Census operations. 


VI. Proposed 2020 Data Products Plan 

1. CSAC recognizes the difficult challenge of weighing the needs of a multitude of user groups and 
appreciates the opportunity to provide feedback on the 2020 data products before the plans 
have been finalized. We commend and recommend continuing the practice of transparent 
deliberations as Census decides what tables to discontinue while considering the tradeoffs 
between accuracy and privacy. 

CENSUS RESPONSE: 

The Census Bureau accepts this recommendation. We will continue to update CSAC and other 
stakeholders on the development of our 2020 Census data products. 


2. CSAC recommends that Census conduct a sentiment analysis of the July 2018 Federal Register 
responses, review internal analytics on the frequency of queries from 2010 tables from 
American Fact Finder, and inquire about analytics from external data visualization and query 
systems, like PolicyMap, Community Commons, IPUMS and NHGIS. As Census makes decisions 
about which tables to include, CSAC recommends that the Census directly engage with 
stakeholder groups who care about the table subject matter for input on which crosstabs are 
most critical for their work at what level of accuracy. In prioritizing additional outreach, CSAC 
recommends that the Bureau review the various sources of input so far and prioritize user 
groups from whom you have not yet heard. We encourage Census to reach out through 
NAC/CSAC for recommendations on specific interest groups, as needed. 

CENSUS RESPONSE: 

The Census Bureau accepts this recommendation. Throughout the remainder of this year, we 
plan to continue conducting outreach and engagement with key stakeholders to determine their 
critical data needs. This feedback, in conjunction with feedback received from the Federal 
Register Notice, will be considered as final data product plans are developed. 


3. CSAC recommends that the Bureau publish a summary of their analysis and/or select 

generalized use cases (policy analysis, legislative, planning, commercial decisions, weighting 
controls, etc.). This will help all users understand the breadth of audiences that the Census 
needs to serve as context when decisions are made to eliminate tables provided in 2010. 

CENSUS RESPONSE: 

A summary of the use cases provided from the Federal Register Notice comment period will be 
published with the final 2020 Census Data Products Plan in a Federal Register Notice. As in 
previous summaries of Federal Register Notice comments, this summary will provide detailed 
counts of the types of comments and use cases we received (e.g., legal, programmatic, etc.) and 
provide examples of use cases that were submitted. 
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4. The December 2018 Federal Register notice stated that the Bureau would consider a design 
change to include the citizenship question on the PL 94-171 Redistricting Data File if the Federal 
Register responses demonstrated a need for them. CSAC would like to know what the response 
to this Federal Register notice was and, subsequently, how it informs Census' final decision 
about the Redistricting Data File. 

CENSUS RESPONSE: 

The Census Bureau is unable to describe the responses to the Federal Register as they have not 
yet been delivered to the Census Bureau from Office of Management and Budget (OMB). The 
Census Bureau had to repost the Federal Register Notice due to an absence of the ability to 
accept comments during the government shutdown. This reposting occurred on February 13, 
2019. OMB is still conducting their review and should deliver those responses to the Census 
Bureau when their review is complete. Those comments will be incorporated with the feedback 
received regarding the prototype redistricting data, which was published in March of 2019 
/ 'https://www2.census.gov/census 2020/) . The consolidated package of comments and 
feedback will be reviewed for indications of needed design changes, if any. 


5. In communicating the need and challenges of implementing differential privacy methods, CSAC 
recommends the Bureau look at lessons from developing the ACS User Guides - providing 
tailored information for each audience with a particular perspective about the implications of 
the changes for their purposes. It would be helpful to frame the guidance regarding the fitness 
of the data for different use cases. 

CENSUS RESPONSE: 

The Census Bureau accepts this recommendation. We will review the lessons learned from 
developing the ACS User Guides, and we intend to provide tailored information to specific 
audiences as appropriate. Additionally, we will continue to use the expertise from our 
Population Division, Center for Enterprise Dissemination and Disclosure Avoidance, and the 
Communications Directorate to create audience-appropriate messaging about the transition to 
differential privacy and the implications for this on the 2020 Census data products to our 
stakeholders. 


6. Beyond the discussion of table selection, CSAC suggests that the Bureau take the 2020 Census as 
an opportunity to expand to new user groups. Census could strengthen non-traditional users as 
new constituencies for census data. For example, national media (New York Times, Wall Street 
Journal, etc.) are active consumers of census data, and they could recommend broader 
associations of data journalists for outreach. Textbook authors or online instructors for data 
science, statistics, or programming are also potential users of census data. These would be 
found through contacts with leading programs at universities. 

CENSUS RESPONSE: 

The Census Bureau accepts this recommendation. As part of our ongoing outreach with data 
users about the 2020 Census data products, we have included academic data users and 
associations, such as the Population Association of America and the American Statistical 
Association. We plan to continue this engagement. We will explore ways we can engage 
textbook authors, online instructors, and data journalists as we continue our outreach. 
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VII. Managing Privacy-Loss Budget for 2020 Census 

1. CSAC thanks the Census Bureau for presenting on this important subject as the Bureau's 
thinking evolves. CSAC understands that exact invariants put strong constraints on the set of 
feasible datasets, making any confidentiality protection method, including the formal privacy 
methods, more vulnerable. This suggests that the privacy harm of releasing any invariant should 
be balanced against the utility of reporting it exactly. While the reapportionment of 
congressional seats may be an invariant that may need to be computed exactly, CSAC 
recommends that the Bureau consider the option of only releasing the exact seat 
apportionment, and noisy state population counts consistent with it, rather than reporting the 
exact state populations needed to compute the reapportionment. 

CENSUS RESPONSE: 

The Census Bureau acknowledges CSAC's suggestion regarding the use of apportionment counts 
as an invariant instead of the use of state level counts. However, after discussions within the 
Census Bureau, we have determined that holding the apportionment counts invariant and 
publishing noisy state level counts would not significantly impact the overall privacy protections. 
Furthermore, we believe that removing the state level Population invariant would be unduly 
confusing to the public and would significantly increase the public education and customer 
engagement requirements associated with the move to formal privacy. As a result, we have 
decided to retain a per state population invariant for the 2020 Decennial Census 

2. Principled balancing of accuracy requirements is an important question. Answering this would 
require collecting data on the common use cases. See discussion under "Proposed 2020 Data 
Products Plan" for several recommendations for collecting and prioritizing common use cases. 
The question of prioritizing use cases is an important one, and the Bureau may consider forming 
a working group to address this. 

CENSUS RESPONSE: 

The Census Bureau accepts CSAC's suggestion regarding the collection and prioritization of use 
cases for the 2020 Data Products Plan. 

At the present time, the Census Bureau has several internal working groups that have been 
meeting for more than a year collecting and prioritizing common use cases for the 2020 Data 
Products and processing the information supplied by users in response to Federal Register 
Notices on these data products. These working groups are now developing suitable accuracy 
metrics. We will expand these working groups to include outside stakeholders in the near future, 
and we will report to CSAC on those activities. 

3. The technical problem of optimizing the mechanism given the outcome of the prioritization, is 
an optimization problem of the type that the Bureau already solves. The other hyperparameters 
in the system, such as the partitioning of the privacy budget across geographical levels, can be 
optimized for the same objective. The Bureau may use previous census data to do this 
optimization, or use recently developed algorithms (https://arxiv.org/abs/1811.07971) to select 
good hyperparameters on the 2020 Census data itself. 
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CENSUS RESPONSE: 


The Census Bureau accepts the suggestion regarding hyperparameters and the use of 2010 data. 
The Census Bureau is already making use of data from the 2010 Census in developing the 2020 
Disclosure Avoidance System. 

We are now also exploring whether we can use changes from 2000 to 2010 to build a model that 
we could then apply to the 2010 data to make predictions about aspects of the 2020 data. Such 
predictions would allow us to increase the accuracy of the 2020 Disclosure Avoidance System 
without impacting the privacy-loss budget. 

Unfortunately, although the optimization mechanism used in the 2020 Disclosure Avoidance 
System has some similarities to other optimization problems that the Census Bureau solves, this 
optimization problem has significant differences both in terms of scale and in terms of the 
number of hyperparameters in the system. As a result, even though overall objective functions 
may have some similarities, the differences remain quite significant. 


4. Statistical analyses on the differentially private datasets would require users to apply a different 
set of techniques. CSAC recommends that the Bureau publish user guides that include examples 
of some of the common kinds of analyses that users of the data typically do. The Bureau may 
consider making available tools to do these kinds of analyses. 

CENSUS RESPONSE: 

The Census Bureau accepts the suggestion regarding the publishing of common kinds of data 
analyses that users typically do, as well as the publishing of tools for performing these kinds of 
analyses. The Census Bureau will evaluate the possibility of making such tools available. 


VIII. Community Partnership and Engagement Program: An Overview and Update 

1. CSAC appreciates the information shared in the presentation. CSAC believes it is crucial to have 
quantitative data on the impact of the partnership program, to inform training and advocate for 
appropriate budget and staffing for future censuses. CSAC therefore encourages Census to 
continue investigation into quantifying the impact of CPEP overall, and Partnership Specialists 
specifically. Perhaps a single metric for impact can't capture the wide variety of target outreach 
populations and outreach efforts across Partnership Specialists—that may suggest a need for 
the development of Partnership Specialist personas and metrics to evaluate success for each 
persona. 

CENSUS RESPONSE: 

Measuring the performance of the Partnership Program is challenging due to the indirect 
relationship between partners and census response rates. Intuitively, it makes sense that 
relationships with trusted voices convey to the American public the importance of completing the 
decennial census and would help increase response rates. Previous evaluations of advertising 
and partnership efforts, however, have found it difficult to attribute which IPC component effects 
on response rates. However, technological changes and new communication vehicles make it 
easier for staff to fine tune efforts to gauge effectiveness. 

For example, staff will use the CRM and other tools to document many of the data points below 
to determine if at least 50 percent of all national partners will have the capacity to reach one or 
more hard-to-count populations by March 2020. 
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The Census Bureau will use the following criteria to help to determine effectiveness of the 
Partnership Program: 


• Governments and partners are aware and support the 2020 Census. 

• Partnerships educate people about the 2020 Census to motivate self-response and 
encourage cooperation with enumerators. 

• Partnership events and activities scheduled in areas with concentrations of low response 
result in increased response rates. 

• Website visits and trends are tracked around the time when partnership community 
events occur. 

• Leveraging partnerships in traditionally low response score areas. 

The Partnership Program will have internal measures to monitor staff work flow and other 
measures to track progress toward objectives. The following measures are examples of metrics 
captured in the CRM that will be used to track the effectiveness: 

• Number of committed State Complete Count Commissions, Complete Count Committees, 
and Tribal Complete Count Committees. 

• Number of organizations with partnership agreements (signed and verbal). 

• Number of organizations engaged by partnership staff. 

• Estimated monetary "value" of activities conducted by partners. 

• Number of partnership activities recorded in the CRM. 

The following are potential methods that the Census Bureau is exploring that could be used to 
capture the metrics. The Census Bureau continues to refine appropriate measures as the 
Partnership Program matures before 2020. 

• Tracking of partners, partner events, number of attendees, and partner provided 
resources. 

• Monitoring of response rates prior to and after events. 

• Partner organizations can use census campaign codes where a link directs people to the 
Census Bureau website electronically and tracks the number of respondents who click 
that link. 

The items listed above are a few of the methods that the Census Bureau is evaluating to 
determine a quantifiable means of measuring the success of the CPEP program overall, and 
partnership specialists individually. 


2. Furthermore, CSAC encourages Census to share lessons about best practices and innovative 
ideas coming out of the Partnership program with the network of Complete Count Committees, 
who are doing similar activities with similar goals. This information should be synthesized and 
considered as input for materials to guide Complete Count Committees and community partners 
for 2030. (FLD) 
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CENSUS RESPONSE: 


In a number of states, State Complete Count Commissions are showing leadership by conducting 
meetings with local government Complete Count Committees (CCCs) or visiting local CCCs to 
ensure that synergy with statewide initiatives occur. 

CCCs at the state and local levels are increasingly requesting Census Solutions Workshops during 
their meetings to help develop work plans. Census Solutions Workshops give CCCs a framework 
for brainstorming that emphasizes bringing diverse partners together and practicing empathy to 
develop relevant solutions for specific hard-to-count communities. 

CCCs are increasingly integrating strong digital engagement strategies into work plans. Digital 
engagement can produce great returns with relatively little effort. By identifying social medial 
influencers in their communities to send out key census messages, they are ensuring that hard- 
to-count communities hear from trusted voices in the electronic realm. 

CCCs understand the role that children play in educating their families about the importance of 
the census, particularly in households with recent immigrants. By appointing top officials of local 
school boards into their committees, CCCs are able to use 2020 Census Statistics in Schools 
materials efficiently and comprehensively once they become available. 

CCCs are harnessing data to ensure that strategies created to reach hard-to-count communities 
address the underlying factors for being hard to count. Committees are using the Response 
Outreach Area Mapper, and are looking to their State Data Centers and Census Bureau Data 
Dissemination Specialists to help get the data and analytics they need to conduct meaningful 
outreach. 


3. To this end, CSAC also recommends that Census work with Partnership Specialists to articulate 
their individualized outreach goals for the 2020 Census and publish those goals to the public. 
Having published goals will help Complete Count Committees understand where outreach 
coverage and outreach gaps exists, to focus their work, and it could inform where we need 
additional Partnership Specialist coverage for the 2030 census. 

CENSUS RESPONSE: 

The Community Partnership and Engagement Program (CPEP) has one primary and two 
secondary goals. By March 2020, the CPEP will secure 300,000 partners that are located in and 
serving the traditionally hardest to count communities. 

Specifically, CPEP will establish partnerships with at least one organization in, or that serves, all 
tracts with a Low Response Score of 30 or higher (9 percent of all census tracts) and in 70 
percent of tracts with a Low Response Score between 20 and 30 (41 percent of all census tracts) 
by March 2020. 

Additionally, CPEP will facilitate the formation of Complete Count Committees (CCCs) or similar 
efforts in all 50 states, the District of Columbia, Puerto Rico, and cities with a population of 
200,000 or more by January 2020. 

Each partnership specialist has an individual goal of securing high quality and high impact 
partners to support the 2020 Census and engage their communities to participate and respond. 
These individual goals are specific to the communities for which they are assigned. These goals 
are further refined and informed by additional data from partner organizations, natural 
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disasters, and or other events that may occur that result in a newly emerged hard-to-count 
population. 


4. In the continued hiring for Partnership Specialists for the 2020 census, CSAC highlights the 
opportunity to continue focusing on populations with the historically greatest undercounts, for 
example young children as highlighted by the representative of Zero to Three in their public 
comment during this meeting, as well as populations Census expects to have increased non¬ 
response in 2020 based on the latest CBAMS data. 

CENSUS RESPONSE: 

The primary focus of the Community Partnership and Engagement Program (CPEP) is outreach to 
populations with historically the highest historic undercounts. Our hiring strategy is to recruit 
partnership specialists who live in the communities in which they will be working. Because of this 
strategy, partnership specialists join the Census Bureau with specific knowledge and language 
skills in these communities. Additionally our partnership specialists are often multilingual, which 
affords us the capability to deliver the right message to the right audiences. In addition to 
community and language skills, they also often have expertise in specific focus areas such as 
early childhood, homelessness, veterans, migrant workers, faith based, differently abled, foreign 
born, and many more. 


5. Regarding the data used to inform Partnership Specialist outreach: the national data at the tract 
level could be misleading in some cases given the ACS 5-year averages and margins of error. 
CSAC encourages Partnership Specialists to reach out to their local civic data and technology 
communities (including local Chief Data Officers/planning offices, community data 
organizations, applied academic centers, volunteer civic tech groups, private sector firms) for 
assistance in using local data and tools to supplement the national data platforms and to 
encourage digital inclusion organizations to get involved in outreach. 

CENSUS RESPONSE: 

The Response Outreach Area Mapper (ROAM) is one of many tools and resources that 
Partnership Specialists can leverage to identify people living in areas that traditionally have 
lower response rates. Partnership Program staff use Census Bureau data and research to 
identify the demographic characteristics of HTC groups. The Census Bureau will distribute this 
information to partners and the public to use when shaping their outreach efforts. In addition, 
regional census centers will use this information when developing and implementing partnership 
activities to illustrate types of partner organizations and where, geographically, to conduct 
events within each region. 

Partnership specialists work closely with partner organizations, State Data Centers, Census 
Information Centers, governmental planning offices, and tribal, federal, and state organizations, 
in addition to community leaders and organizations, to inform them where populations that are 
traditionally undercounted are and to ascertain the best outreach strategies that resonate with 
these communities. 


Page 15 


IX. Current Population Survey Annual Social and Economic Supplement 

CSAC thanks the Census for the presentation and discussion of improvements in the CPS ASEC 
supplement. The user community will benefit from the improved accuracy. As discussed during the 
session, the communication and outreach related to the changes is critical for the user community's 
utilization of the data and understanding of the new process. The following are CSAC's 
recommendations related to the questions Census presented: 


1. In both the refined processes for the economic and health data of the CPS ASEC, CSAC agrees it 
represents a break in time series for being able to compare the newly processed data with 
legacy data. As such, the communication of the change is critical for the user community. 
Overall, CSAC recommends Census consider defining a 'break in time series' and develop a 
protocol for procedures to follow in such an occurrence, such as setting best statistical practices 
for bridging the transition and communicating the changes to users. 

CENSUS RESPONSE: 

The Census Bureau accepts this recommendation. We will establish a taskforce through our 
Current Population Survey Annual Social and Economic Supplement (CPS ASEC) Change Control 
Board to develop a proposed protocol for procedures to follow when there is a break in a time 
series. This protocol will be shared with the Quality Program Staff, which maintains the quality 
standards and guidelines for the entire Census Bureau. Since ideally this protocol should apply to 
all time series estimates throughout the Census Bureau, the protocol would need to be approved 
by the Census Bureau's Methodology and Standards Council. 


2. The current outreach being conducted by the Census regarding the change in the CPS ASEC is 
encouraged and appreciated. This includes multiple published papers, press releases, 
attendance and communication at relevant user conferences, and hosting an expert user group 
for a presentation. In addition to these efforts, CSAC recommends development of a user 
matrix that can cross reference the communication methods for each user group. For example, 
'user group 1' was notified of these changes through x,y,z, etc. 

CENSUS RESPONSE: 

The Census Bureau accepts this recommendation. 


User Group 

Expert 

Meeting/ 

Webinar 

Website 

Pre- 

Release 

Webinar 

Working 

Papers/ 

Research 

Files 

America 

Counts 

Story 

Conference 

Presentations 

Media 


X 

X 


X 


Academics/ 

Researchers/ 

Students 

X 

X 


X 


X 

General public 


X 



X 


Other federal 
statistical 
agencies/ 

X 

X 


X 


X 
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data users 







Respondents 





X 


Congressional 

offices 

X 

X 


X 




3. CSAC mentioned the following additional outreach suggestions: 

a. A short article that groups like the Association of Public Data Users or the American 
Statistical Association could place in their newsletters 

CENSUS RESPONSE: 

We expect that our "America Counts" story on the change will be suitable for use in 
newsletters for these types of groups. 

b. Academic Community - communication to students and professors that use the data for 
research 

CENSUS RESPONSE: 

We have presented (and will be presenting) papers discussing the impact of the changes 
at various academic conferences, including the Population Association of America, 
APPAM (Association for Public Policy Analysis and Management), Joint Statistical 
Meetings of the American Statistical Association, and the Society for Government 
Economists. 

c. Policy research organizations - as participants in the expert meeting and a source for 
additional communication suggestions. 

CENSUS RESPONSE: 

At the May 13,2020, meeting, we will ask participants to make suggestions for 
additional outreach regarding the changes. In response to a request by one participant 
invited to the May 13 expert meeting, we will be providing a call-in and webinar option 
for the expert meeting. Approximately 20+ people will attend in person and 13 people 
have responded that they will be calling in to the meeting. The webinar will be recorded 
and available on census.gov. 

d. State and Local Public Health Officials 
CENSUS RESPONSE: 

We will be doing some research on networks of state and local public health officials in 
order to send out the" America Counts" story with links to the working papers on the 
impact on health insurance estimates to these networks. 
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e. Population Association of America 
CENSUS RESPONSE: 

We presented a half-day pre-conference workshop on the upcoming changes to the CPS 
ASEC on April 9,2019, in Austin, Texas. 

f. Potential use of social media and/or blogs, including for lay audiences 
CENSUS RESPONSE: 

We are currently writing a story for "America Counts" that should be an excellent vehicle 
to reach lay audiences. 

Since our CSAC presentation, we have launched a new page on census.gov that provides 
users with easy access to all the research and data sets related to the redesign of the 
processing system. https://www.census.aov/data/datasets/time-series/demo/income- 
Dovertv/cDS-asec-desian.html . 


4. CSAC recommends Census investigate the potential use of administrative records as a method 
of quality control of responses to the income questions. Can administrative records be used in 
cases where respondents are resistant to answer specific questions about income, as opposed 
to follow up and options to enter a range of income? 

CENSUS RESPONSE: 

The Census Bureau accepts this recommendation. The Census Bureau is actively engaged in 
research regarding the potential uses of administrative records to evaluate the quality of 
responses to the income questions and to reduce respondent burden. As an example, the 
following presentation was given at the PAA pre-conference workshop on using administrative 
records to evaluate the accuracy of respondents taking advantage of the bracket option for 
income responses: "Are Bracket Responses Accurate." It is available at 
httDs://www.census.aov/librarv/workina-DaDers/2019/demo/SEHSD-WP2019-22.html. 

X. Economic Census Internet Self-Response Experience 

1. CSAC appreciated the presentation on the Internet self-response experience for the Economic 
Census. There are many potential advantages to an all-electronic approach, including greater 
accuracy, cost savings, greater convenience for a majority of respondents, and overall efficiency. 
Having a single place for businesses to manage all their surveys is another positive as it will lead 
to consistent experiences and clear expectations. The single customer relationship management 
tool should help eliminate duplicate efforts to address concerns. 

CENSUS RESPONSE: 

We appreciate your feedback and support for this initiative. 

2. The quality of the implementation of the self-response experience is critical for success. Having 
one platform for the development and delivery of surveys should make it easier to make 
necessary improvements as needed. 
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CENSUS RESPONSE: 

We appreciate your feedback and support for this initiative. 


3. With respect to security and registration, CSAC recommends following the latest NIST guidelines 
available at https://paRes.nist.gov/800-63-3/sp800-63b.html 

The guidelines do not recommend periodic changes of passwords. CSAC recommends the 
implementation of multi-factor authentication to increase security. CSAC also recommends 
educating respondents on phishing risks. 

CENSUS RESPONSE: 

We have been analyzing the system paradatafrom the registration process. Based on this 
review, we understand that respondents are struggling mainly with password complexity, set up 
and usage of security questions, and password resets. We have briefed Census Bureau 
management on the paradata findings, as well as the recommendations of CSAC. We plan to 
meet with our Office of Security very shortly to work on possible solutions, including the 
investigation of multifactor authentication. 


4. CSAC recommends following human-centered approaches to the design of self-response 

instruments such that the effectiveness, efficiency, learnability, memorability, and utility of the 
user interface is optimal in a wide variety of platforms (e.g., mobile). For example, additional 
unnecessary clicks or page loads will likely decrease response rates. There are also opportunities 
to increase responses by providing value to respondents. Perhaps the same portal used to 
answer surveys could be used to deliver customized products to businesses (e.g., regional 
reports, or reports on a particular sector of the economy). Tutorials (e.g., videos) could also be 
used to help, for example, small business owners with lower computer literacy. Monitoring non¬ 
response rates should enable the identification of types of businesses that may be struggling 
with the new approach, which could inform redesigns of the system and/or the availability of 
more resources. 

CENSUS RESPONSE: 

For the 2017 Economic Census, we performed respondent debriefings and are using that 
feedback to improve our instruments for the 2018 Company Organization Survey and Annual 
Survey of Manufactures (COS/ASM). We also provided some tutorials on the Respondent Portal 
and the Centurion tools for the 2017 Economic Census. We are adding more tutorials for our 
annual 2018 COS/ASM collection to better assist respondents with the reporting process. We are 
currently investigating using the Respondent Portal to provide more tailored messaging to our 
respondents regarding data products and are considering doing this when they first burn their 
Authentication code as well as a thank you for response. When considering major changes to 
the instruments, we also perform usability testing with respondents to help inform the best 
design options. 


5. CSAC also recommends continued attention paid to addressing employee turnover issues (i.e., 
employee responding to surveys leaves the company) and encouraging respondents to 
designate delegates. 


Page 19 



CENSUS RESPONSE: 


We would like respondents to use the delegation feature in the tool to share survey access within 
their company. If they fail to do this, we have ways to delegate for them or provide them with a 
new authentication code. We do know based on usage statistics and debriefings that many 
respondents do not know that delegation is an option. We are planning to make enhancements 
to the system so this functionality will be more obvious to respondents. 


XI. Public Comments 

CSAC appreciates the public's interest in and support of the Census Bureau and the time taken to submit 
public comment to the CSAC. As a response to the public comments, CSAC would like to hear the 
following from the Census Bureau: 


1. In light of the comments from CBAMS presentation on possible unwillingness of some 
populations to respond to the 2020 Census, as well as the March 2019 Harvard study that 
estimated an increased undercount of the Hispanic population from adding the citizenship 
question, CSAC would like to know the justification for the statement in Section 8 of supporting 
statement Part A of the 2020 Census Information Collection Request (ICR) that "The Census 
Bureau has identified no credible quantitative evidence that the addition of a citizenship 
question would impact the net undercount of the 2020 Census." 

CENSUS RESPONSE: 

The CBAMS results are referring to the census response rate-whether someone self 
responds by Internet or telephone. This is not the same as the coverage rate-whether 
someone is included in the final census counts. Lower self-response rates do not mean all 
the non-responders will be missed but that the Census will cost more and that we will have 
to visit more homes in person to count the residents and collect their information in person. 


2. CSAC agrees that the undercount of young children is a critical issue and would like a discussion 
of it included in the evaluation plans session recommended in the above section responding to 
the "Update on the 2020 Census". We would also appreciate a summary in your written 
response of the actions taken in the 2020 census procedures, materials, and outreach to reduce 
the undercount. 

CENSUS RESPONSE: 

The Census Bureau is planning to brief the CSAC on these issues in their fall 2019 meeting. 
Current actions in 2020 Census procedures, materials, and outreach are still in final 
development, but include revised questionnaire wording, help screens, 
interviewer/enumerator training, and frequently asked questions, as well as integrated 
messaging throughout the media campaign and specific messaging and outreach to target 
groups. 
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APPENDIX 


May 31, 2019 

Research & Testing and 2020 Census Evaluation, Experiment, and Operational Assessment Study Plans/Reports 


2020 Census Program for Evaluations and Experiments (CPEX) 1 

> Evaluations (7) 

Administrative Record Dual System Estimation 

Evaluation of the Reengineered 

Address Canvassing Operation 

Research on Hard to Count Populations: Non-English Speakers and Complex Household Residents including Undercount of Children ["language"] 

Analysis of Census Internet Self-Response Paradata by Language 

Evaluating Privacy and 

Confidentiality Concerns 

The Undercount of Young Children: A Qualitative Evaluation of Census Materials and Operations: Part 2, PES and AdRec Match to 2020 Census 

Group Quarters Advance Contact (GQAC): Refining Classification of College or University Student Housing 

> Experiments (5) 

Real-Time 2020 Administrative 

Record Census Simulation 

Extending the Decennial Census 

Environment to the Mailing 

Materials 

Evaluation of the Optimization of Self-Response in the 2020 Census 

Vacant Crowdsourcing Experiment 

Impact of Asking Citizenship on Self-Response Rates, Data Quality, and Coverage Assessment in the 2020 Census 

2020 Census Operational Assessments z (48) 

Archiving Operational Assessment 

Census Questionnaire Assistance Operational Assessment 
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Content and Forms Design Operational Assessment _ 

Coverage Improvement Operational Assessment _ 

Count Question Resolution Operational Assessment _ 

Decennial Logistics Management - Logistics Management Support Operational Assessment _ 

Decennial Logistics Management - Space Acquisition and Lease Management Operational Assessment _ 

Decennial Service Center _ 

Demographic Analysis Operational Assessment _ 

Enumeration at Transitory Locations Advance Contact Operational Assessment _ 

Enumeration at Transitory Locations Operational Assessment _ 

Evaluation and Experiments Operational Assessment _ 

Federally Affiliated Count Overseas (FACO) Operational Assessment _ 

Field Infrastructure - Field Office Administration and Payroll Operational Assessment _ 

Field Infrastructure - Recruiting, Onboarding, and Training Operational Assessment _ 

Forms, Printing and Distribution Operational Assessment _ 

Geographic Partnership Programs Operational Assessment _ 

Group Quarters Advance Contact Assessment Report _ 

Group Quarters Enumeration and Military Enumerations Assessment _ 

In-Field Address Canvassing Operational Assessment _ 

In-Office Address Canvassing Operational Assessment _ 

Integrated Partnership and Communications Operational Assessment _ 

Integrated Partnership and Communications Contract Assessment _ 

Research to Support the Integrated Partnership and Communications Program _ 

Internet Self-Response Operational Assessment (Internet & Mail Contact) _ 

Island Areas Censuses Operational Assessment _ 

Language Services Operational Assessment _ 

Local Update of Census Addresses (LUCA) Operational Assessment _ 

Maritime Vessel Enumeration Report _ 

Non-ID Operational Assessment _ 

Nonresponse Followup Operational Assessment _ 

Paper Data Capture Operational Assessment _ 

Post Enumeration Survey (PES) SAMPLING and ESTIMATION Operational Assessment _ 

Post Enumeration Survey (PES) FIELD OPERATIONS Initial Listing (IL) and Initial Housing Unit Followup (IHUFU) Operational Assessment 

Post-Enumeration Survey (PES) FIELD OPERATIONS Person Interview (PI) & Person Followup (PFU) Operational Assessment _ 

Post-Enumeration Survey (PES) FIELD OPERATIONS Final Housing Unit Followup (FHUFU) Operational Assessment _ 

Post-Enumeration Survey (PES) MATCHING Initial Housing Unit (IHU) Matching Operational Assessment _ 

Post-Enumeration Survey (PES) MATCHING Person Matching Operational Assessment_ 
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Post-Enumeration Survey (PES) MATCHING Final Housing Unit (FHU) Matching Operational Assessment 

Redistricting Data Program Operational Assessment 

Response Processing Operational Assessment 

Self-Response Quality Assurance Operational Assessment 

Service-Based Enumeration Assessment Report 

Systems and Applications in the 2020 Census (Security, Privacy, and Confidentiality) 

Update Enumerate Operational Assessment 

Update Leave Operational Assessment 

Response Rates Assessment Study 

Item Nonresponse Rates Assessment Study 

Quality Control Study Plan for Listing Operations 

2020 Census Quality Control (QC) 

Quality Control Study Plan for Enumeration Operations 

Address Canvassing QC Results 

Update Leave QC Results 

Nonresponse Followup QC Results 

Person Interview QC Results 

Independent Listing QC Results 


1 As study plans undergo review by the Decennial Research Objectives and Methods (DROM) working group, some evaluations or 
experiments may be de-scoped from the research program. 

2 This is an inventory of 2020 Census operational assessments as of May 31, 2019. All 2020 Census operations have at least one operational 
assessment planned, except for Data Products and Dissemination, Systems Engineering and Integration, and IT Infrastructure. In addition 
for the Program Management operation, a "quality assessment" will be produced for each business process in lieu of an operational 
assessment. They will be managed and tracked by the Decennial Program Management Office. 
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